Fine-Grain Distributed Shared Memory on Clusters of Workstations

Shared memory, one of the most popular models for programming parallel platforms, is becoming ubiquitous both in low-end workstations and high-end servers. With the advent of low-latency networking hardware, clusters of workstations strive to offer the same processing power as high-end servers for a fraction of the cost. In such environments, shared memory has been limited to page-based systems that control access to shared memory using the memory’s page protection to implement shared memory coherence protocols. Unfortunately, false sharing and fragmentation problems force such systems to resort to weak consistency shared memory models that complicate the shared memory programming model. This thesis studies fine-grain distributed shared memory (FGDSM) systems on networks of workstations to support shared memory and it explores the issues involved in the implementation of FGDSM systems on networks of commodity workstations running commodity operating systems. FGDSM systems rely on fine-grain memory access control to selectively restrict reads and writes to cache-block-sized memory regions. The thesis presents Blizzard, a family of FGDSM systems running on a network of workstations. Blizzard supports the Tempest interface that implements shared memory coherence protocols as user-level libraries. Therefore, application-specific protocols can be developed to eliminate the overhead of the fine-grain access control. First, this thesis investigates techniques to implement fine-grain access control on commodity workstations. It presents four different techniques that require little or no additional hardware (software checks with executable editing, the memory’s controller ECC, a combination of these two techniques, and a custom fine-grain access control accelerator board). Furthermore, the thesis examines the integration of hardware fine-grain techniques within commodity operating systems. Second, this thesis investigates messaging subsystem design for shared memory coherence protocols. It explores implications of extending Berkeley active messages so that explicit polling is not required and protocols are not limited to request/reply semantics. The thesis explores supporting implicit polling using binary rewriting to insert polls in the application code. In addition, it shows that shared memory coherence protocols, while not pure request/reply protocols, have bounded buffer requirements. Accordingly, it proposes a buffer allocation policy that does not require buffers in a node’s local memory in the common case, yet is robust enough to handle arbitrary traffic streams. Third, this thesis investigates extending FGDSM systems to clusters of multiprocessor workstations. FGDSM systems are especially suited for supporting shared memory on multi-